Pacific Rim Objective Measurement Symposium (PROMS) 2016 Conference Proceedings by Quan Zhang

Author:Quan Zhang
Language: eng
Format: epub
Publisher: Springer Singapore, Singapore

Types of answers

Score

Fully correct code and produces the expected solution definition meeting all the specifications.

3

Mostly correct code towards the progression of actual solution definition meeting most of the specifications

2

Partly correct code towards the progression of actual solution definition meeting some of the specifications

1

Not attempted

0

The professional judgment of experts ensures the test items or tasks are relevant and representative of the construct domain (Cavanagh, 2009). Expert screening of items on the CS1 instrument was obtained from four CS1 lecturers, two of them currently teaching and the other two are doing their Ph.D. research in various universities of Australia, who had over five years of experience in teaching CS1. The assumed difficulty hierarchy matched with the experts’ ratings except for one question, which was subsequently replaced. Based on the advice, the visual illustration of some questions [1(d) & 5(d)] was improved, some questions were rephrased, and some were re-written.

5. The scaling model. The scaling model chosen was the Rasch Partial Credit model using the Rasch Unidimensional Measurement Model (RUMM 2030) computer program (Andrich, Sherridan, & Luo, 2010). The Partial Credit Model can scale a combination of dichotomous and polytomous responses.

6. Item technical quality. The item technical quality involves aspects such as unambiguous phrasing, accurate answer keys, and suitable reading levels for the target population (Messick, 1996). This was ensured by a small pilot study of the 20 items with 10 students selected from APU, Malaysia. The pilot study indicated issues such as some questions not eliciting response consistent with the expected difficulty hierarchy, wording misunderstanding, and non-completion of all the questions. For example, students found (a) and (b) part of Question One to be of same difficulty level, similarly, (a) and (b) part of question Two had the same fault, and, so these questions were revised and rewritten. The analysis of the responses also showed that Question 3(b), 4(a) and 5(a) were badly worded and in order to avoid confounding the respondents, 3(b) and 4(a) were replaced, and 5(a) was re-phrased. Similarly, Question 3(d) was replaced as well because no one had attempted it, which was taken as a signal that this question may not contribute to any level of ability estimation in the final administration. The instrument was originally written using Java but was then translated to C++ and Python to accommodate the languages taught in the targeted institutes.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.